Relational Reinforcement Learning

Reinforcement learning [10] is a subtopic of machine learning that is concerned with software systems that learn to behave through interaction with their environment and receive only feedback on the quality of their current behavior instead of a set of correctly labelled learning examples. Although reinforcement learning algorithms have been studied extensively in a propositional setting, their usefulness in complex problems is limited by their inability to incorporate relational information about the environment [9]. Relational Reinforcement Learning is concerned with reinforcement learning in domains that exhibit structural properties and in which different kinds of related objects exist. These domains are usually characterized by a very large and possibly unbounded number of different possible states and actions. In this kind of environment, most traditional reinforcement learning techniques break down. This thesis discusses the development of a first applicable relational reinforcement learning (or RRL) system [6,7]. This RRL system combines Q-learning with the representational power of relational learning by using relational representations for states and actions and by employing a relational regression algorithm to approximate the Q-values generated through a standard Q-learning algorithm. Due to the use of a more expressive representation language to represent states, actions and Q-functions, the proposed relational reinforcement learning system can be potentially applied to a wider range of learning tasks than conventional reinforcement learning. It also enables the abstraction from or parametrization of specific goals or even of specific learning environments and allows for the exploitation of results from previous learning phases when addressing new (more complex), but related situations. Relational representations also permit the use of structural information or the existence of objects and relations between objects in the description of the resulting policy (through the learned Q-function approximation). The relational regression algorithm used in the RRL system generalizes over learning examples with a continuous target value and makes predictions about the value of unseen state-action pairs, using a relational representation for both the learning examples and the resulting function. The RRL setting places a number of constraints on the regression techniques that can be used. Learning data generated by a Q-learning algorithm is incremental (i.e., the regression algorithm should be able to both use and update its current model) and it supplies the regression algorithm with a moving target function, as the estimates of Q-values of examples are mostly random at the start of learning but become more reliable with more experience. Because of the relational representation used for states and actions, the algorithm cannot treat the set of all examples as a vector space. Although in all practical applications the dimension of the state space will be finite, it will not be known at the start of the learning experiment and may even vary during the experiment (e.g., when the number of objects in the agent’s environment varies). Three incremental relational regression algorithms are developed that can be used in the RRL system. The TG algorithm incrementally builds first order re-